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PERFORMANCE  MEASURES  MATRDC 


INTRODUCTION 

The  objective  of  this  study  was  to  deiermiitt  the  utility  and  validity  of  existing  Air 
Force  job  peiformance  measures  used  to  evaluate  the  perfonnance  of  ainnen.  This  repon 
contains  the  results  of  our  investigation  of  the  existing  and  emarging  criteria,  peiformance 
measures,  and  other  indicators  of  job  performance  in  the  operational  Air  Force. 


Baekgrotmd 

The  Job  Performance  Measurement/Enlistment  Standards  (JPM)  Project,  initiated  in 
1980,  is  a  joint-Service  research  program  exploring  TnethnH!t  for  assessing  the  job 
performance  ct^tability  of  enlisted  personnel.  As  part  of  this  project,  the  Air  Force  Hunuui 
Resources  Laboratory  (AFHRL)  has  developed  and  administered  different  types  of 
performance  measures  to  first  term  airmen  in  various  occupations.  The  developed  measures 
in  combinati<»  ccxistimte  the  Job  Peiformance  Measureixtent  System  (JPMS).  These  efforts 
support  the  ctmclusion  found  in  the  literature  that  hands-on  testing,  although  the  most  valid 
form  of  measuring,  is  not  always  feasible  or  practical  because  of  cost,  safety,  and  other 
factors.  Due  to  its  inqnacticality,  hands-on  test  can  not  be  operationalized  in  most  situations, 
therefore,  they  will  serve  as  benchmarks  by  which  surrogate  measures  can  be  assessed.  Four 
types  of  job  perfonnance  measures  (i.e.,  hands-<Hi  tests,  interview  tests,  rating  forms,  and 
knowledge  tests)  were  develqied  and  administered  to  selected  enlisted  specialties  as  pan  of 
the  JPM  project 

AFHRL,  in  cooperatitxi  with  the  Perfomance  and  Technology  Division  of  the  Defense 
Training  and  Peiformance  Data  Center  (TPDQ,  is  developing  a  Triuning  Technology  Data 
Base  (TTDB).  The  TTDB  will  link  task,  technology,  and  perfonnance  dimensions  through 
development  of  job  perfonnance  data  and  measurement  methodology.  The  link  will  be 
established  so  as  to  capitalize  cm  existing  data  sources  which  are  cmnpatible  with  the 
developed  performance  data  and  methodology.  The  TIDB  will  allow  potential  users  of  the 
Air  Force  JPMS  to  determine  the  job  performance  measures  most  suited  to  their  particular 
application.  A  natural  follow-on  to  this  work  is  the  identification  of  existing  perfonmance 
measures  in  the  qterational  Air  Force  and  to  validate  these  identified  measures  using  the 
results  of  tte  JPM  project  The  products  of  this  study  will  provide  a  common  body  of 
knowledge  about  those  existing  and  emerging  job  perfornuuice  measurement  technologies  that 
have  the  potential  to  support  DoD  training. 


MtUutd 


The  parameters  and  systems  in  the  matrix  were  developed  based  on  a  review  of  the 
literanue,  completed  as  a  part  of  the  fint  phase  of  this  project,  and  interviews  with 
knowledgeable  Air  F(»ce  personnel. 

An  interview  protocol  was  created  based  on  the  information  required  to  complete  the 
matrix.  This  protocol  was  loosely  followed  during  the  field  interviews.  The  interview 
protocol  is  included  in  this  report  as  Appendix  A. 

This  task  is  focused  on  five  specialties:  4S4X0  Aerospace  Propulsion  Specialist 
(Engines).  455X2  Communication  and  Navigation  System  Specialist,  454X1  Aerospace 
Ground  Equipment  Mechanic  (AGE),  324X0  Precision  Measuring  Equipment  Labcnratory 
Specialist  (PMEL),  and  122X0  Aircrew  Life  Suppmt  Specialist  (ALS).  These  Air  Force 
Specialties  (AFSs)  are  five  of  the  eight  specialties  cm  which  the  JPMS  procedures  were 
developed.  The  other  three  specialties  were:  492X1  Communications  Systems  Radio 
Operator,  272X0  Air  Traffic  Control  Operator,  and  732X0  Personnel  Specialist  For  purposes 
of  comparison  between  the  results  of  task  and  the  JPMS  data,  we  selected  AFSs  that  were 
includ^  in  the  JPMS  efforts. 

We  attetiq)ted  to  speak  widi  individuals  at  all  levels  of  the  Air  Force  in  order  to  get  a 
broad  perspective  of  what  is  being  developed  and  what  is  (^)eiational.  Interviews  were 
conducted  with  Air  Staff  (HQ  USAF)  Functional  Managers  at  the  Pentagon  for  the  five 
specialties  of  interest.  The  possibility  of  operadtHial  performance  indicators  were  discussed 
and  the  functional  managers  supplied  points-of-ccmtact  for  MAJCOM  Functional  Managers, 
and  contacts  at  HQ  Strategic  Air  Cmnmand  (SAQ,  Military  Airlift  Command  (MAC),  and 
Tactical  Air  Coirunand  (TAC),  HQ  Air  Force  Military  Perstmnel  Center  (AFMPC)  and  HQ 
Air  Force  Training  Command  (ATQ. 

Interviews  were  then  conducted  at  Randolph  and  Brooks  AFBs.  Researchers  met  with 
the  training  managers  at  Air  Training  Comrruuid  for  the  five  specialties  of  interest.  On-the- 
Job  Training  (OJT),  Core  Autoinated  Maintenance  System  (CAMS)  and  other  procedures  that 
might  be  used  to  document  performance  were  discussed.  Interviews  were  also  conducted  with 
training  and  classification  perscmnel  at  AFMPC.  Discussions  were  held  with  personnel 
involved  in  develt^ment  of  the  Advanced  On-the-Job  Training  System  (ACTS),  currently 
renamed  the  Base  Training  Systeoa.  The  acronym  AOTS  is  used  to  indicate  this  system  in  the 
remainder  of  this  report  The  research  staff  interviewed  CAMS  system  developers,  where 
CAMS  as  well  as  other  automated  systems  (Personnel  Evaluation  and  Analysis  Program 
(PEAP),  Automated  Maintenance  Standardization  and  Evaluation  Program  (SALLY),  and 
(polity  Assurance  Tracking  and  Trend  Analysis  System  (QANTTAS)  were  discussed. 

The  research  staff  then  visited  four  Air  Fence  bases  (2  MAC,  1  TAC,  1  MAC/TAC)  to 
interview  service  members  at  the  operational  level  in  all  five  AFS  of  interest  The  interviews 
consisted  of  questions  based  on  the  interview  protocol  and  a  discussion  of  day  to  day 
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activities.  Approximately  four  airmen  or  nonctxnmissioned  officers  (NCOs)  were  interviewed 
in  each  of  the  five  specialties.  At  three  installations,  interviews  were  conducted  with  the 
Quality  Assurance  (QA)  personnel  on  the  evaluations  that  QA  conducts.  While  at  two  of  the 
bases,  we  spoke  to  the  Training  Managers. 


Organization  of  the  Report 

The  next  section  of  this  report  describes  the  development  of  the  matrix.  The  subsequent 
section  of  the  report  describes  the  perfornaance  measures  found  at  the  (q)erational  level 
followed  by  a  secti<»  which  describes  the  systems  found  at  the  (^)erational  level.  The  final 
section  summarizes  the  report  and  presents  our  conclusions.  The  sumnoary  presents  the 
completed  matrix. 
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MATRIX  DEVELOPMENT 


HumRRO  staff  members  developed  a  matrix  to  summarize  information  on  the 
properties  of  peifonnance  information  cuirently  available  in  the  operational  Air  Force.  We 
accomplished  this  by  identifying  and  accessing  infoimadon  on  the  performance  measurement 
characteristics  of  existing  job  performance  measures.  The  noatrix  provides  information  to 
answer  the  following  kinds  of  questions  about  performance  information: 

1.  What  information  is  included  in  the  parameter  (e.g.,  units  of  production, 
mistakes,  stnties)? 

2.  What  type  of  information  is  included  in  the  parameter  (e.g.,  subjective, 
objective,  behavioral)? 

3.  What  is  the  level  of  aggregation  (e.g.,  individual,  unit,  organization,  AFS)? 

4.  Where  is  the  information  recorded  or  where  is  it  obtained  (e.g.,  written 
documentation,  AF  forms,  maintenance  records)? 

5.  How  is  the  information  measured  (e.g.,  observation,  self-report)? 

6.  How  is  the  informadon  currently  used  (e.g.,  documentation,  determine 

maintenance  needs,  determine  training  needs)? 

7.  How  could  the  information  be  used  for  other  purposes  (e.gM  making  human 
resource  decisions)? 

8.  What  is  the  quality  of  the  information  (e.g.,  reliability,  validity)? 

9.  How  could  JPMS  information  be  helpful  to  the  field  in  making  better/odier 

human  resource  decisions  (e.g.,  selection  and  classification,  training,  program 
evaluation,  and  identification  of  individual  training  deficiencies)? 

The  parameters  and  systems  in  the  matrix  are  based  on  a  review  of  the  literature, 
completed  as  a  part  of  die  fim  phase  of  this  project,  and  interviews  with  knowledgeable  Air 
Force  persormel.  We  developed  a  preliminary  matrix  based  (xi  interviews  with  Air  Force 
personnel  at  the  Headquarters  and  program  levels.  We  then  refined  the  matrix  based  on 
interviews  at  the  (qierational  level  The  kinds  of  performance  measures  searched  for  are 
described  below. 


4 


Job  Performance  Measures 


The  matrix  can  include  the  Job  Performance  Measurement  System  (JPMS)  procedures, 
as  well  as  the  operational  Air  Force  measurement  procedures  we  uncovered.  The  JPMS  were 
developed  for  the  Joint-Service  Job  Performance  Measurement  Project.  Several  procedures 
were  employed  in  the  JPMS  such  as  walk-through  performance  tests  (WTPT),  job  knowledge 
tests  (JKT),  and  rating  forms.  These  procedures  are  described  below. 

WTPT.  The  WTPT  is  a  combination  of  die  hands-on  performance  test  (HOPT)  and 
the  interview  techruque.  Hands-on  tests  are  the  highest  fidelity  measures  of  acmal  job 
proficiency  because  the  HOPT  measures  performance  while  actually  executing  a  work  sample. 
However,  these  tests  are  expensive  to  develop  and  administer.  The  HOPT  are  used  as 
benchmarks  against  which  other  measures  are  evaluated.  Surrogate  measures  are  the 
interview,  JKT  and  others  such  as  performance  ratings.  The  WTFT  blends  aspects  of  the 
work  sampling  used  in  HOPT  and  the  observo^  interview.  In  the  interview  the  examinee  talks 
through  the  procedures  necessary  to  perform  the  task.  The  addition  of  the  interview 
component  ^ows  for  the  measurement  of  critcal  tasks  that  are  too  expensive,  time 
consuming,  or  dangerous  to  access  in  the  hands-on  rxxxle.  The  hands-on  component  is  used 
for  those  tasks  which  can  be  performed  safely  without  being  too  costly  in  time  and  resources 
and  can  be  performed  without  damaging  the  equipment  and  components.  In  the  interview 
component  the  examinee  uses  the  "show  and  tell"  approach,  explairting  how  the  task  is 
performed  through  words,  gestures,  and  demonstrations.  Interview  testing  takes  place  in  the 
work  setting  so  the  examinee  can  point  to  components  and  tools. 

JKT.  The  JKT  was  developed  to  measure  on-the-job  performance  and  to  provide  a 
surrogate  for  the  labor  intensive  and  more  expensive  WTPT.  The  JKTs  are  multiple-choice 
paper-and-pencil  tests  designed  to  measure  the  exatxtinee’s  knowledge  of  job  teclmical  content 
and  procedures.  Test  items  are  based  on  the  task  oriented  content  of  WTPT  tasks. 

Rating  Forms.  Rating  forms  are  for  job  poformance  assessment  by  the  supervisor,  by 
the  examinee,  and  the  examinee’s  peer.  The  rating  forms  are  task-specific,  dimensional, 
global,  or  service-wide.  Task  and  dimensional  rating  forms  rate  technical  proficiency  on 
specific  tasks  and  dimensions  relating  to  a  specific  AFS.  Global  rating  forms  are  used  to 
assess  general  technical  and  interpersonal  proficiencies.  Service-wide  forms  are  used  to  rate 
proficiency  in  areas  deemed  important  for  success  in  the  Air  Force.  Rating  forms  may  also 
be  used  to  assess  the  level  of  motivation  of  the  examinee  at  the  time  of  testing. 

Information  was  collected  on  existing  job  performance  systems  for  comparison  to  the 
JPMS.  The  kinds  of  information  collected  during  the  data  collection  phase  are  included  in 
the  following  matrix  parameters. 
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Matrix  Parameters 


The  perfortnance  measurement  paiametos  are  shown  in  Figure  1.  The  parameters 
listed  are  the  rows  of  the  matrix.  The  second  dimension,  the  columns  of  the  matrix,  are  the 
job  performance  systems  reviewed.  The  taxonomy  has  twelve  paratneters  which  fall  within 
three  broad  areas:  content,  performance  measurement,  and  evaluation.  Also,  two  utility 
parameters  are  included  that  relate  to  current  and  other  possible  uses  of  the  job  performance 
data.  The  parameters  are  described  below. 

Content.  Content  parameters  relate  to  the  physical  aspects  of  the  data,  the  domain  and 
participants. 


Accessibility  of  data.  Job  performance  data,  to  be  useful  as  a  pan  of  the 
validation  of  selection  criteria,  must  be  accessible.  The  matrix  indicates  the  ownership  and 
location  of  the  data,  how  the  data  are  stored,  and  how  the  data  may  be  accessed. 

Domain.  The  domain  includes  the  mission,  occupational  specialty,  and  echelon 
(e.g.,  individual,  workcenter,  unit,  organization). 

Participants.  The  main  thrust  is  job  performance  measurement  of  the 
individual.  This  parameter  can  include  rank,  time  in  service,  and  other  infonnation  pertairung 
to  the  individual. 

Performance  measurement.  Performance  measurement  overlaps  with  content  and 
evaluation  parameten,  as  well  as  four  mutually  exclusive  parameters.  These  are  the  item  or 
function  being  measured,  the  associated  data,  the  procedures  used,  and  the  method  for 
recording  the  data. 

Item  measured.  The  item  or  function  on  which  the  job  performance  measure  is 
taken.  This  may  include  a  procedure,  units  of  production,  number  of  mistakes,  or  successful 
sorties,  or  time  to  failure  of  system  following  repair. 

Associated  data.  An  informational  parameter  for  any  other  data  associated  with 
the  performance  measure. 

Procedure.  The  procedure  is  die  method  for  obtaining  the  performance  data. 
The  procedure  may  be  observation,  paper-and-pencil  records,  or  electronic  transmission. 

Recording  method.  The  method  used  to  record  the  original  data.  The  method 
may  be  a  check  list,  special  form,  standard  AF  form,  or  from  previously  recorded 
maintenance  records. 

Evaluation.  Parameters  in  the  evaluation  category  deal  with  the  quality  of  the  job 
performance  data. 
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Type  of  i/tformation.  The  matrix  indicates  whether  the  data  are  subjecdve, 
objecdve,  or  behavio^. 

Research  questions.  This  indicates  the  intended  purpose  of  the  recorded  data. 
Is  it  for  training  effecdveness,  unit  readiness,  promotion,  etc.? 

Fidelity.  This  parameter  includes  reliability  and  validity.  Reliability  is  the 
degree  of  consistency  between  items  from  the  same  domain.  Validity  is  the  degree  to  which 
the  job  performance  measurement  instrument  accurately  measures  w^  it  is  supposed  to 
measure.  We  attempted  to  determine  if  die  data  collected  on  job  performance  systems  have 
indicators  of  reliability  and  validity. 

Timing  of  data  collection.  We  looked  at  timing  issues.  Timing  issues  include 
announced  and  unaimounced  assessment  procedures,  timing  of  job  performance  measurement 
in  relation  to  the  career  progress  of  individuals,  timing  within  a  training  session,  timing  in 
relation  to  a  broader  training  cycle,  and  timing  in  relation  to  the  individual’s  Service  career. 

Sampling  method.  The  sampling  method  was  determined  during  data 
collection.  The  sampling  method  indicates  how  participants  are  selected. 

Utility.  Utility  relates  to  the  end  produa  and  its  uses. 

Current  use  of  data.  How  the  collected  data  is  currently  used. 

Other  possible  uses  of  data.  Other  potential  future  uses  of  the  data  collected 
(e.g.,  research,  promotion,  training  assessment). 
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OPERATIONAL  MEASURES 


Opeiadonal  measures  found  in  existing  and  planned  Air  Force  systems  are: 

Apprentice  Knowledge  Test  (AKT) 

Career  Development  Course  (CDC) 

Enlisted  Performance  Report  (EPR) 

Job  Qualification  Standa^  (JQS) 

OJT  Supervisor  ratings 

Quality  Assurance/Quality  Control  (QA/QC)  evaluadons: 

Personnel  Evaluadons  (PE) 

Quality  Verificadon  Inspection  (QVI) 

QVI  Level 

Specialty  Knowledge  Test  (SKD 
Each  of  the  c^teradonal  measures  is  discussed  below. 


AppmOee  Knowk^  Tut  (AET) 

The  AKT  was  developed  as  a  tool  to  identify  individuals  who  can  bypass  technical 
training  school  and  go  direcdy  to  an  assigned  unit  AKTs  are  100  item,  multiple-choice, 
domain-based  tests.  They  include  declarative,  general  aptitude,  specialty-specific  equipment 
and  job-relevant  knowlet^  needed  to  perfbnn  at  die  3-leveL  An  AKT  is  developed  first  by 
SMEs  who  prepare  an  outline  of  the  specialty  knowledge  required  prior  to  tm-the-job 
assignment  in  an  AFS.  The  outline  components  are  weighted  to  determine  the  number  of 
items  to  be  develtqied  for  each  component  The  SMEs  and  Air  Force  Occupation 
Measurement  Center  (OMC)  test  developers  follow  a  dynamic,  iterative  process  to  develop 
test  items.  All  items  have  four  response  qptitms.  After  items  are  develq)ed,  die  test  is 
administered  to  the  same  SMEs,  who  then  review  the  items  for  accuracy  and  clarity.  The 
OMC  test  developers  also  review  die  final  product  The  qualifying  scores  for  selected  AKTs 
are  based  on  the  performance  of  technical  school  graduates. 


Catur  DtvtkpmnU  Count  (CDC) 

The  CDC  is  a  study  course  and  testing  procedure  primarily  associated  with  skill  level 
advancement  The  courses  are  prepared  for  specific  AFSs  and  as  an  integral  pan  of  die  OJT 
program.  The  end-of-course  exam  must  be  successfiiUy  completed  before  an  airman  can  be 
upgraded  to  the  next  higher  skill  level.  The  subjects  trf  the  course  are  those  things  an  airman 
needs  to  know  about  their  occupational  specialty  at  their  career  level.  The  course  involves 
self  study  materials.  The  supervisor  or  an  assigned  trainer  works  with  the  ainnan  during  the 
study  course.  Study  is  at  the  job  site  and  in  off  dufy  hours.  The  course  is  modular  and  may 
have  five  or  more  volumes.  There  is  a  test  for  each  volume  administered  at  the  job  site  wiA 
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immediate  feedback  to  the  participant  When  all  volume  testing  has  been  successfully 
completed  the  airman  goes  to  a  designated  testing  center  for  the  end-of-course  exam. 


Enlisted  Performance  Report  (EPR) 

The  EPR  is  an  annual  performance  review  for  each  enlisted  person.  Also,  there  is  a 
provision  for  a  mid-term  review  and  feedback.  The  airman’s  overall  job  perfOTmance  is  rated 
on  a  Eve  point  scale.  The  EPR  emphasizes  the  "total  person",  rather  dian  job  performance 
alone.  It  is  said  to  be  less  inflated  than  the  previous  system  (Airman  Perfonnance  Report 
(APR))  which  was  based  on  a  nine  point  scide. 


Job  QuaUficadon  Standards  (JQS) 

The  JQS  axe  the  tasks  and  standards  associated  with  a  particular  AFS.  An  aimum 
must  denunstrate  the  ability  to  perform  each  of  the  tasks  and  be  certified  on  each  one.  The 
JQS  may  include  several  levels  of  Specialty  Task  Statements  (STS)  such  as: 

Air  Fdtce  Job  Qualification  Standards  (AFJQS) 

Command  Job  Qualification  Standards  (CJQS),  and 

Unit,  work  center,  dr  duty  position  requirements  and  standards. 

AFJQS  are  Air  Force  wide  standards.  CJQS  are  command  level  standards:  standards  in 
addition  to  AFJQS  required  by  the  MAJCOM  or  installation  comnumd.  Unit,  work  center, 
and  duty  position  standards  are  additional  requirements  at  the  local  level. 

When  a  new  person  comes  into  a  unit  the  supervisor  or  an  assigned  trainer  will  go 
through  the  JQS  task  list  to  determine  which  tasks  the  airman  needs  to  be  trained  on.  If  the 
supervisor  determines  that  upgrade  training  is  needed  the  aimoan  goes  to  the  Field  Training 
Detachment  (FTD)  for  the  training.  After  the  FTD  training  the  airman  learns  to  perform  the 
tasks  associated  with  the  job  through  OJT.  When  both  the  supervisor  and  the  airman  agree  it 
is  time,  the  supervisor  will  observe  the  airman  perfmming  the  task.  The  satisfactory  or 
unsatisfactory  performance  result  is  entered  on  the  task  evaluatitm  form.  Air  Force  Form  803, 
and  the  form  is  put  in  the  OJT  jacket  Recording  die  satisfactory  performance  on  the  task 
means  the  airman  is  certified  to  do  the  task. 
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OJT  Supervisor  Ratings 


OJT  supervisor  ratings  are  made  by  bodi  supervisors  and  commanders  based  on  review 
of  the  OJT  records  and  observed  performance.  The  ratings  art  usually  projected,  satisfactory, 
or  unsatisfactory  and  include  the  date  of  the  rating.  These  ratings  are  periodic  and  scheduled 
evaluations  to  determine  if  an  individual  is  making  adequate  progress  on  his  or  her  OJT 
program. 


Quality  AssuruncelQuaUty  Control  (QAIQC)  Evaluations 

Personnel  assigned  to  QA/QC  duties  conduct  several  types  of  investigations  both 
announced  and  unannounced.  Two  types  of  investigations  are  associated  with  evaluations  of 
personnel.  One  is  the  Personnel  Evaluation  (PE)  which  involves  observation  of  on-the-job 
performance.  The  observation  may  involve  over-tiie-shoulder,  interview,  hands-on  or  a 
combination  of  techniques  chosen  by  the  evaluator. 

The  other  type  is  the  Quality  VerificatitMi  Inspection  (QVI).  Although  the  QVI  is  an 
inspection  of  equipment,  the  result  of  the  inspection  is  associated  with  an  individual  who 
performed  or  was  involved  in  the  maintenance  or  repair.  Passes  and  failures  and  the 
individuals  involved  are  a  pan  of  the  records  maintained  by  the  QA/QC. 

Quality  Assurance  is  under  the  Deputy  (^omnoander  of  Maintenance  (DCM).  An 
individual  firom  the  QA  ofitice  visits  the  maintenance  shops  to  conduct  the  inspection.  The 
results  of  QA/QC  evaluations  are  discussed  witii  the  supervisor.  The  supervisors  are  not 
given  a  copy  of  the  inspections.  The  record  of  the  evaluations  are  maintained  by  the  QfiJQC. 
The  QA/(^  prepare  periodic  reports  of  findings.  The  reports  to  the  work  units  do  not 
include  names  of  individuals. 

At  some  commands  there  are  QA  people  assigned  within  the  unit  This  person 
conducts  the  day-to-day  QA  inspections.  This  information  is  not  recorded  in  any  standardized 
way. 


QVIUvel 

The  QVI  Level  is  used  exclusively  in  die  PMEL  specialty  in  conjunction  with  the 
PMEL  Automated  Maintenance  System  (PAMS).  Because  of  the  criticality  and  precision  of 
the  work,  PMEL  has  a  more  stringent  quality  assurance  system  than  the  rest  of  maintenance. 

Process.  There  are  four  QVI  Levels.  The  QVI  Level  determines  the  percentage  of 
items  per  level  that  get  checked.  Each  person  has  a  designated  level.  The  percentage 
check^  at  each  level  are  as  follows: 


11 


OVI  Uvd  Percent  Items  Inspected 

1  Coin  50% 

2  Coin  25% 

3  Coin  12.5% 

4  Coin  6.25% 

The  word  "coin"  comes  firmn  the  original  system  in  which  one-cent  coins,  contained  in  a  four 
quadrant  shaker,  were  actually  used  to  determine  whether  the  item  was  checked.  The 
probability  from  the  "coin"  system  is  now  programmed  into  the  PAMS  system.  When  the 
Quality  Assurance  (QA)  person  is  entering  the  paperwork  into  PAMS,  the  systems  alerts 
him/her  if  the  person  should  be  inspected. 

When  a  new  perstm  joins  the  unit  d»  first  five  items  woriced  cm  are  inspected,  and  if 
all  five  are  passes,  the  airman  enten  at  a  3*coin  level  The  QA  person  inspects  100%  of  the 
paperwork  that  goes  in  on  each  item  to  be  sure  it  is  all  filled  out  conectly  and  to  be  sure  an 
inordinate  amount  of  time  was  not  spent  on  dm  item.  He  then  scans  the  item  into  the  PAMS 
system.  It  is  here  that  the  system  aim  dm  QA  person  if  it  is  time  for  a  QVI  inspection. 

The  QA  perscm  inspects  dm  equipment,  if  dmre  is  a  fiulure  the  airman  will  move  into 
a  position  called  4R.  The  next  four  items  are  pulled  and  if  dmre  are  any  failures,  the  airman 
is  dropped  down  to  level  3R  and  dm  next  4  items  ate  checked  dmre  and  so  on.  If  there  were 
no  failures  at  dm  4R  level  the  airman  moves  on  to  4*.  At  4*,  dm  items  are  inspected  at  the 
tegular  coin  4  rate,  6.25%,  but  if  any  of  the  next  fourteen  inspections  are  failures,  the  airman 
will  be  dropped  to  dm  3R  level  In  order  to  move  tq>  to  dm  next  level  an  airman  must  pass 
18  consecutive  inspecticms  at  dm  noniml  inspection  cycle. 

Because  dm  inspections  ate  based  on  chance,  it  is  possible  that  an  airman  can  go  for 
many  months  without  an  inspection.  In  order  to  avoid  diis,  it  is  built  into  the  QVI  system 
that  an  airman  can  only  have  31  uninspected  items  and  the  next  item  is  iiupected. 

There  is  a  second  type  of  inspection  called  over-dm-shoulder  inspection  where  a 
person,  rather  than  the  equipment,  is  inspected.  A  failure  of  an  over-the-shoulder  can  result 
in  the  dropping  of  a  level 

The  QVI  Level  can  be  an  indicator  of  job  performance.  However,  the  QVI  Level 
needs  to  be  used  in  conjunction  widi  ocher  factors.  The  QVI  Level  can  relate  to  the  type  and 
kinds  of  equiprrmnt  selected  for  calibration  or  repair.  Production  rates  will  reflect  the  type 
equipnmnt  worked  on.  Also,  those  who  select  harder  work  are  mme  likely  to  get  a  failure, 
while  those  who  play  it  safe  widi  easier  items  are  likely  to  have  less  failures. 
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Specialty  Knowle^e  Test  (SKT) 


Where  SKTs  are  available,  they  are  used  to  assess  the  airman’s  knowledge  relating  to 
job  tasks.  The  SKT  is  a  written  multiple  choice  test  which  airmen  take  when  they  become 
promotable.  It  primarily  tests  knowledge  recall  although  some  items  are  performance 
oriented.  Tests  are  scheduled  and  announced  in  advance.  This  allows  airmen  to  study  and 
prepare  for  the  test  The  SKT  score  comprises  (xie-fifth  of  the  Weighted  Airman  Promotion 
System  (WAPS)  score  used  to  select  airmen  for  promotion. 

The  SKT  is  developed  by  Subject  Matter  Experts  (SMEs)  primarily  fnmi  the  CDC. 
SMEs  also  use  material  from  regulations,  text  books,  and  other  sources.  There  are  two  levels 
of  the  SKT:  (a)  one  for  promotion  to  E5  and  (b)  tnie  for  promotion  to  E6  and  E7.  The  E6 
and  E7  tests  cover  the  same  basic  material.  However,  since  several  years  pass  between 
promotion  from  E6  to  E7,  several  revisions  are  made  to  the  SKT  by  the  time  the  airman  takes 
the  test  for  promotion  to  E7.  The  SKT  is  revised  annually,  items  are  reordered,  and  the 
answer  key  changed. 
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OPERATIONAL  SYSTEMS 


The  automated  systems  in  use,  being  implemented  and  planned  by  the  operational  Air 
Force  are: 


Core  Automated  Maintenance  System  (CAMS) 

Advanced  On-the-Job  Training  System  (AOTS) 

Personnel  Evaluation  and  Analysis  Pmgnm  (PEAP) 

Quality  Assurance  Tracking  and  Trend  Analysis  System  (QANTTAS) 
Autonoated  Maintenance  Standardization  and  Evaluation  Program,  SALLY 
PMEL  Automated  Maintenance  System  (PAMS) 

Aerospace  Life  Support  Mam^ement  System  (ALSMS) 


Con  AutomaUd  MaUUtmmet  Syitom  (CAMS) 

The  CAMS  is  an  Air  Force  directed  project  to  improve  management  and  utilization  of 
maintenance  resources  through  use  of  automated  data  processing  for  logistic  informadon. 
CAMS  was  designed  to  support  all  base-level  aircraft,  ground-launched  cruise  missiles, 
engines,  trainers,  support  equipment,  test  equipment,  missiles,  munitions,  and 
communicadons-electronics  maintenance.  The  Training  Management  subsystem  aiul  the 
Quality  Assurance/Quality  Control  subsystem  are  discussed  below. 

Training  ManagemetU  Subsystem 

The  Thuning  Management  subsystem  of  CAMS  was  designed  to  extend  the  capability 
of  training  management  personnel  by  reducing  thdr  administradve  burden.  The  CAMS  data 
base  includes  required  training  courses;  inspection,  special  qualificadons,  and  On-the-J(^ 
Training  (OJT)  i^ormatkm;  training  status,  due  and  completed  dates;  and  an  individual’s 
complete  training  history.  This  data  base  can  be  used  to  determine  and  validate  training 
needs.  CAMS  iaq)leiiients  OJT  requirements  in  AF  Reguladon  50-23.  The  Training 
Management  subsystem  is  described  in  Air  Force  Manual  66-279,  Vol  XVn. 

CAMS  is  operational  at  most  major  maintenance  locations.  Woricstadcms  are  in  place 
at  workcenter  locations  and  are  being  used.  CAMS  will  not  have  a  central  data  base.  The 
data  bases  are  ctmtrolled  at  or  below  the  MAJOOM  level.  Codes  for  the  data  are  not 
compatible  across  MAJCOMs.  The  data  bases  are  cm  a  mainframe  at  the  installation  level. 
Modules  in  CAMS  will  vary  reflecting  MAJCOM  and  installation  requirements. 
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There  are  no  local  data  bases.  Local  units  are  tied  into  installation  or  higher  level  data 
bases.  Units  can  access  data  from  the  data  bases  based  on  their  level  of  access  code.  Each 
person  has  a  unique  ID  number  assigned  at  the  installation  level.  There  is  a  cross  reference 
of  ID  to  SSN.  An  operator  with  an  appropriate  access  code  may  access  data  by  name  or 


SSN. 


The  performance  indicators  included  in  the  Training  Management  subsystem  of  CAMS 
are  discussed  below. 

Job  Qualification  Standards  (JQS).  CAMS  provides  a  meduxl  of  recording  the 
progress  of  an  airman  during  OJT.  CAMS  can  be  u^  as  an  auttnnated  version  of  the  OJT 
Jacket  The  task  titles  (STS  listing)  along  widi  study  references  are  recorded  in  the  system. 
When  an  airman  denaonstrates  he  or  she  can  perform  a  task,  a  code  and  date  are  entered  into 
CAMS  (^posite  that  task  statement  In  other  words,  instead  of  circling  a  task  statement  on 
the  STS  listing  in  the  OJT  Jacket  an  operator  types  in  die  code  (e.g.,  *0  and  the  date  (e.g.,  07 
DEC  88)  on  a  screen  using  CAMS.  In  additi(»  to  the  task  number,  title,  code,  and  date  the 
following  are  included: 

Name 

Enqiloyee  number 

Qrganizadon.  number,  kind,  and  type 
Workcenter  mnemonic  and  numbtt 
Date  assigned 
Initial  evaluation  date 
Evaluation  exemption  indicator 
Date  of  last  evaluaiicm 
Upgrade  training  (UGT)  status  and  date 
Immediate  supervisor  name  and  number 

The  task  listing  in  CAMS  can  include: 

Air  Force  Qualificadmi  Standards  (AFJQS), 

Command  Job  Qualification  Standards  (CJQS),  and 
Unit,  workcenter,  or  duty  position  specific  qualification 
requirenoents  and  standards. 

This  system  is  used  by  the  training  manager  and  the  workcenter  supervisor  to  monitor  the 
progress  of  personnel  in  training.  It  replaces  the  manually  maintained  STS  and  AF  Fmm  797. 
OJT  information  can  be  viewed  using  a  variety  of  screens;  by  individual,  by  task  statement, 
by  workcenter,  etc. 
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Career  Development  Course  (CDC).  CAMS  contains  data  on  the  CDC  participation 
for  each  individual  by  name  and  employee  number.  It  includes: 

Course  code,  sequence  number,  and  volume  number 
Volume  Stan  date,  score  and  completkm  date  (open  book 
practice  exercise-  not  a  test) 

Course  exam  (GB)  status  (c  «  completion;  f  -  failed) 

CE  completion  daso  and  score 
CE  retest  data 

Apprentice  Knowledge  Test  (AKT).  CAMS  includes  the  AKT  score. 

Supervisor  and  Commander  Ratings.  CAMS  contains  both  supervisOT  and 
commander  evaluations.  Hie  data  contains  evaluation  indicators  (0  «  projected;  1  » 
satisfactory;  2  «  unsatisfactory)  and  evaluation  dates. 

Quality  AssuraneelQuatity  Control  Subsystem 

The  Quality  Assurance/Quality  Control  (QAA^  subsystem  is  described  in  Functional 
Description  FD-G83-004  Supplement  VIIB,  August  1989.  This  subsystem  is  currently  "on 
hold."  Wofk  oa  the  subsystem  is  scheduled  for  the  coming  year.  Tte  QA/QC  subsystem, 
like  the  other  CAMS  modules,  will  not  be  standardized  throughout  the  Air  Force.  It  will  be 
MAJCOM  specific.  Data  bases  will  be  at  the  base  level. 

A  locally  approved  form  will  be  used  to  record  die  results  of  equipment  inspections 
and  personnel  evaluadcros.  The  evaluations  are  classified  by  type:  Completed  Maintenance 
Action  (CMA),  Coopleted  Maintenance  Inspection  (CMI),  Cor^leted  Supervisory  Inspection 
(CSI),  Supervisory  Evaluation  (SE),  Task  Evaluation  (TE),  Evaluator  Proficiency  Evaluation 
(EPE),  and  Quality  Verificatiem  Inspection  (QVI).  The  results  of  the  inspections  and 
evaluations  are  entered  into  CAMS.  Also,  the  system  allows  entry  of  the  results  of  special 
inspections  and  evaluations. 

A  CMA  is  an  after-the-fact  evaluation  of  a  previously  conqileted  maintenance  action 
performed  on  an  aircraft  or  equipment  items.  A  C^  is  an  after-die-fiia  evaluation  of  a 
previously  conpleted  maintenance  inspectitm  of  aircraft  or  equipment  items.  A  CSI  is  an 
after-the-foct  evaluation  of  a  supervisor  who  has  performed  and  documented  an  inspectitm  of 
a  completed  maintenance  action  on  aircraft  or  equipment  items.  An  SE  is  an  over-the- 
shoulder  evaluatitm  of  a  supervisor  during  actual  task  performance.  A  TE  consists  of  an 
over-the-shoulder  evaluation  of  a  maintenance  techniciiui  during  actual  job  performance.  The 
EPE  serves  as  a  certification  process  for  people  performing  evaluations.  The  QVI  is  an 
inspection  of  an  item  or  piece  of  equipment 
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The  QA  inspector  is  required  to  document  discrepancies  and  provide  a  rating  for  the 
evaluation.  Entries  into  CAMS  may  include: 

Transaction  type  (L  «  load;  C  •  change;  D  *  delete) 

Date,  shift,  time,  workcenter 

Name,  grade,  control  number,  aiul  type  evaluation 

Task  or  job  description 

Task  code 

The  categ(»y  or  class  of  discrepancy 
Whether  the  discrepancy  is  minor  ot  major 
Type  of  errors  made 
Summary  statement 

The  QA/QC  subsystem  will  provide  for  botii  user  defined  reports  and  automated 
reports.  User  defined  reports  will  be  available  by  name,  task,  wotkMnter,  or  other  selected 
combinations.  Reports  can  show  type  of  evaluation,  equipment  aiul/or  task,  rating,  date  of 
evaluation,  and  next  due  date  for  any  required  evaluation.  Also,  historical  reports  will  be 
available.  CAMS  will  provide  a  list,  sorted  by  a  specified  mnemonic  code,  of  all  assigned 
personnel  and  their  persminel  evaluation  histc^  for  a  user  defined  time  period;  or.  a 
personnel  evaluation  history  covering  a  specified  time  period  for  a  specific  employee/name. 

An  automated  evaluation  report  is  planned.  The  automated  system  will  contain  a 
matrix  table  for  loading  weighted  or  baseline  requirements.  The  overall  rating  of  the 
evaluation  will  be  automatically  calculated  when  the  number,  type,  ot  category  of 
discrepancies  are  entered.  The  system  will  automatically  update  training  subsystem  course 
codes  and  job  qualification  stawUuds  when  an  entered  evaluation  report  satisfies  a  related 
periodic  inspection  requirement  Unsatisfactory  coaq)letion  of  an  automated  inspection  repon 
will  automatically  sch^ule  a  follow-on  evaluation  within  a  user-defined  time  period. 


Advanced  On-Uu-Job  Training  System  (AOTS) 

The  Advanced  On-the-job  Training  System  (AOTS),  currently  renamed  the  Base 
Training  System,  was  developed  to  test  a  design  concept  for  improving  cunent  practices  in 
the  Air  Force  On-the-Job  Tritining  (OJT)  program.  It  is  a  computer-based,  training 
management  training  develqmient  and  delivery,  and  training  evaluation  system  designed  to 
enhance  OJT  activities  in  the  operational  setting.  ACDHTS  was  designed  to  be  in  strict 
compliance  with  Air  Force  OJT  standards  and  AFR  50-23.  The  system  focuses  on  job 
position  qualifications,  whereas  OJT  focuses  on  task  qualifications. 

Woric  began  on  AOTS  in  1985  at  Bergstrom  AFB,  Texas.  The  development  team  was 
AFHRL,  Douglas  Aircraft  Company,  and  Ball  Systems  Engineering  Division.  The  prototype 
system  involved  over  600  Active,  ^serve,  and  National  Guard  personnel  in  five  specialties 
for  one  year.  AOTS  is  being  expaiuled  to  cover  other  AFSs  and  other  installations. 
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The  expansion  plan  calls  for  a  fully  networked  personal  computer  (PC)  based  system. 
Other  possibilities  are  a  stand-alone  PC  based  system  or  a  mainframe  based  systeoL  llie  user 
level  workstation  configuration  would  be  based  on  user  needs.  The  workstation  may  include 
a  printer,  optical  maik  reader,  interactive  video  disks,  or  any  other  required  equipment.  The 
installation  system  manager  will  have  access  to  file  space  to  store  data  bases  and  will  have 
access  to  long  haul  communications  networks.  The  Air  Force  and  Functional  Area  Manager 
or  MAJCOM  system  administrator  will  have  similar  equipment  as  the  installation  system 
manager.  The  development  site  will  have  all  the  hardware  equipment  used  anywhere  in  the 
system. 


ACTS  has  three  components:  Managements  Evaluation,  and  Training. 

Management  Component 

The  trainer,  evaluator  or  supervisor  uses  the  Training  Scheduler  editor  in  the  Management 
Component  to  schedule  or  forecast  training  or  evaluation  events.  There  are  four  events; 

Knowledge  Training 
Knowledge  Evaluation 
Performance  Training 
Performance  Evaluation 

The  Training  Scheduler  allows: 

1.  Trainers  to  schedule  Knowledge  Training  and  Performance  Training  events 

2.  Evaluators  to  schedule  Knowledge  Evaluation  and  Performance  Evaluation  events 

3.  Supervisors  to  schedule  all  four  types  of  events 

The  Reports  and  Notices  pan  of  the  Management  Component  permits  both  individual 
and  summary  reports  of  all  training  and  evaluation  results. 

Evaluation  Component 

The  Evaluation  Component  allows  access  to: 

1.  Knowledge  test  items  (true/false  and  multiple  choice  questions) 

2.  Performance  evaluation  products  (oral  test  guides  and  performance  evaluation 
checklists. 

Knowledge  test.  Knowledge  test  can  be  administered  both  on-line  and  off-line.  Test 
administered  on-line  are  automatically  scored.  A  test  administered  off-line  is  scored  by  use 
of  an  optical  mark  scanner  answer  sheet  or  by  manually  entering  the  responses  to  each 
question  using  a  keyboard.  The  system  checks  the  off-line  test  results  against  the  knowledge 
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test  answer  key  file  and  accepts  pass/fail  results  for  each  perfonnance  step  and  the 
performance  test  as  a  whole. 

Up  to  75  copies  of  an  off-line  test  may  be  printed  in  response  to  a  single  authorized 
request  from  a  worlutation.  The  test  control  number  will  appear  on  each  copy.  A  knowledge 
test  answer  key  will  also  be  printed  if  requested  at  the  dme  that  the  test  are  printed. 

Petformance  evaluation.  The  evaluation  component  supports  pre-training,  post¬ 
training,  and  quality  control  evaluations.  All  performance  evaluations  are  administered  off¬ 
line.  Scoring  is  accomplished  by  using  an  optical  scanner  check  list  sheet  or  by  manually 
entering  the  "pass"  or  "fail"  ratings  for  each  step  in  the  check  list  The  evaluations  are 
automatically  scored  by  the  system.  No  answer  key  is  available  or  printed  for  a  performance 
test. 


The  periodic  selection  for  a  quality  control  evaluation  event  includes  the  task  to  be 
evaluated,  the  airman  to  be  evaluate^  and  two  external  evaluators.  An  automatic  process 
identifies  the  airman  to  be  evaluated  and  the  external  evaluator  candidates  for  each  task 
identified  for  evaluation.  The  Management  component  of  the  system  notifies  the  QC 
Administrators  of  tasks,  evaluatees,  and  evaluates  identified  as  candidates  for  training  quality 
control.  The  system  supports  manual  selection  of  tasks,  evaluatees,  and  evaluators  for 
evaluations. 

When  the  results  of  the  evaluations  are  entered  using  optical  mark  readers  or  the 
keyboard,  the  system  automatically  scores  the  results.  When  a  QC  evaluation  is  failed,  a 
notice  to  the  commander  recommending  decertification  on  the  task  is  generated. 

Training  Component 

The  training  component  contains  the  computer  assisted  instruction  and  interactive 
video  disk  instruction. 


Personnel  Evaluation  and  Analysis  Program  (PEAP) 

The  Personnel  Evaluation  Analysis  Program  (PEAP)  is  a  micro-computer  aided  trend 
identification  and  analysis  program  developed  for  use  by  aircraft  maintenance  quality 
assurance/deficiency  analysis  personnel.  It  was  tteveloped  for  the  quick  identification  and 
accurate  analysis  of  negative  maintenance  performance  trends.  PEAP  was  developed  by  the 
Air  Force  Logistics  Management  Center  (AFLMC)  using  dBASE  m.  PEAP  not  only 
identifies  negative  personnel  and  equipment  analysis  performance  trends,  but  also  provides  the 
data  needed  to  determine  the  causes  of  those  trends. 

Background.  The  Air  Force  saw  a  need  to  shift  the  emphasis  of  aircraft  maintenance 
quality  assurance  (QA)  from  one  that  centers  on  the  individual  who  fails  to  meet  quality 
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standards  to  one  centered  on  finding  the  specific  causes  contributing  to  evaluation  failures. 
Consequently,  the  AFLMC  developed  PEAP,  a  real-time,  on-line  performance  trend 
identification  and  analysis  system.  This  original  version  was  fielded  by  TAG  and  Alaskan  Air 
Command  (AAC)  in  January  1988.  MAC/LGM  expressed  an  interest  in  PEAP  and  had 
AFLMC  develop  a  modified  version.  The  Final  Report  on  this  effon  stated  that  HQ 
MAC/LCM  implemented  MAC  PEAP  command-wide  on  1  Oct  88.  Air  Force  Reserves 
(AFRES)  associated  with  MAC  units  in  the  Fourth  and  Fourteenth  Air  Forces  also 
implemented  MAC  PEAP  at  this  time. 

Function.  There  are  four  categories  of  quality  assurance  inspection/evaluations: 

1.  Personnel.  Evaluation  of  a  maintenance  action  performed  by  the  aircraft  technician  or 
supervisor. 

2.  Technical.  Inspection  of  equipment  following  a  maintenance  inspection  or  repair 
action. 

3.  Special.  Inspection  initiated  by  higher  headquarters,  a  weapons  system  manager,  or 
the  Deputy  Coirunander  for  Maintenance  (DCM).  This  inspection  may  be  conditional 
or  procedural  compliance  oriented. 

4.  SafetylUnsafe  condition.  Unsafe  act  by  an  individual  (Detected  Safety  Violation)  or  an 
unsafe  condition  in  which  maintenance  personnel  are  not  directly  involved  (Unsafe 
Condition  Report). 

Each  event  is  also  assigned  a  Type  of  Event  Code  (TEC)  which  details  what  kind  of  work 
was  inspected.  TECs  are  assign^  by  base  level  QA  managers.  The  first  two  digits  usually 
correspond  with  the  Work  Unit  Code  (i.e.,  23  »  Engines).  The  third  digit  denotes  the  specific 
event  (i.e.,  23A  s  Engine  final  major  maintenance). 

There  are  three  phases  to  PEAP: 

1.  Initial  InspectionlEvaluation.  Data  is  entered  onto  Form  30  in  PEAP  database. 

2.  Analysis.  PEAP  provides  nine  analysis  reports  citing  trends  in  TECs,  work  centers, 
equipment,  etc.  Two  of  these  reports  relate  to  personnel  evaluation: 

a.  Complete  Personnel  Evaluation  Summary.  PEAP  displays  a  summary  of  every 
personnel  evaluation  QA  has  performed  in  the  last  12  months,  soned  either 
alphabetically  or  numerically  on  the  name  or  employee  number,  respectively  of 
the  personnel  evaluated. 

A  sample  report  form  is  shown  here: 
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Evaluation  History  by  (name)  Cutoff:  (75%)  Jul  1,  1968  to  Sept  1988 


NAME 

EMP 

NO _ 

TOTAL 

PASS 

PASS 

RATE 

CUTOFF 

FLAG 

AVG 

SCORE 

SCORE 

BATING 

HUBBLE 

00664 

2 

1 

50.0 

** 

85.0 

S 

HUGGINS 

00103 

1 

0 

00.0 

*  * 

70.0 

U 

JONES 

00028 

2 

2 

100.0 

100.0 

0 

MCDANIELS 

02002 

1 

1 

100.0 

94.0 

0 

b.  Single  Individual  Evaluation  History  Report.  Displays  totals,  pass  rates,  and 
average  scores  for  the  different  types  of  QA  evaluations  performed  on  a  single 
individual  in  the  last  12  months.  A  sample  report  form  is  shown  here: 


Name:  HUBBLE 

Emp  No: 

00664 

Pass 

Rate:  50.0% 

Avg 

Score : 

85.0% 

Adj 

Pass 

Rate:  50.0% 

Adj 

Avg 

Score: 

85.0% 

TYPE 

INSP 

TOTAL  OUT 

EXC 

$AT 

MAR 

UNSAT 

N/RAT 

TE 

0 

0 

0 

0 

0 

0 

0 

CMA 

0 

0 

0 

0 

0 

0 

0 

CSI 

2 

1 

0 

0 

0 

1 

0 

CMI 

0 

0 

0 

0 

0 

0 

0 

SE 

0 

0 

0 

0 

0 

0 

0 

EPE 

0 

0 

0 

0 

0 

0 

0 

DSV 

0 

0 

0 

0 

0 

0 

0 

UCR 

0 

0 

0 

0 

0 

0 

0 

••TOTAL**  2  1  0  0  0  1  0 


3.  Inspection  Redirection  Reports.  Reports  generated  here  are  used  to  measure  and 
evaluate  QA  inspector  productivity  and,  if  necessary,  redirea  inspection  efforts. 


Quality  Assurance  Tracking  and  Trend  Analysis  System  (QANTTAS) 

The  purpose  of  the  Quality  Assurance  Tracking  and  Trend  Analysis  System 
(QANTTAS)  is  to  help  aircraft  maintenance  Quality  Assurance  (QA)  personnel  keep  track  of 
important  infonnation.  It  is  an  information  tracking  and  trend  analysis  microcomputer 
package.  QANTTAS  is  intended  to  serve  as  a  prototype  and  as  an  interim  procedure  until  a 
standard  system  is  in  place.  The  QA/QC  subsystem  envisioned  for  CAMS  should  replace  this 
and  any  other  non-standard  program  in  use.  liie  system  was  developed  by  the  Air  Force 
Logistics  Management  Center  (AFLMC).  The  sponsoring  organization  is  HQ  United  States 
Air  Force  in  Europe  (USAFE)/LG  with  HQ  I^icific  Air  Forces  (PACAF)  and  HQ  AAC  as 
cosponsors.  Therefore,  the  procedures  are  tailored  to  Tactical  Air  Forces  (TAF)  policies. 
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Background.  Since  1984  there  have  been  substantial  changes  in  MAJCOM  policies 
on  the  QA  function.  Rather  than  route  each  inspection  form,  findings  are  collected  and 
submir  ,d  w*  •'.  /  in  order  to  reveal  negative  performance  trends.  TTie  QA  managers  were  to 
conduct  trena  analysis  to  uncover  negative  porfonnance  indicators.  This  put  a  heavy  cracking 
and  data  analysis  burden  on  QA  managers.  TAC  asked  AFLMC  to  help  in  preparing  an 
automated  trend  identification  and  analysis  tool.  The  response  was  the  Personnel  Evaluation 
Analysis  Program  (PEAP)  which  was  fielded  by  TAC,  AAC.  Air  Force  Systems  Command 
(AFSC),  and  Air  Force  Reserves  AFRES  Tenth  Air  Force.  A  version  was  also  created  for 
MAC  with  their  differences  in  QA  policies  incorporated.  This  MACPEAP  was  fielded  by 
MAC  and  AFRES  Fourth  and  Fourteenth  Air  Forces. 

USAFE  was  not  involved  with  PEAP,  but  as  a  TAF  organization,  its  policies  changed 
in  step  with  those  of  TAC.  The  creation  of  the  Product  Improvement  Branch  (PIB)  added 
another  task  for  the  QA  personnel  of  compiling  deficiency  report  information.  Because  of 
this  and  the  other  additional  burdens,  QA  personnel  soon  began  building  automated  tools  to 
aid  in  keeping  track  of  these  various  functions.  HQ  was  unconofoitable  with  this  due  to  the 
uncertainty  of  the  accuracy  and  reliability  of  these  products,  as  well  as  the  difficulty  they 
would  have  in  inspecting  and  understantfing  each  of  these  tools. 

QANTTAS  was  developed  to  standardized  the  automated  QA  system  across  the 
command.  It  is  admitted  it  is  an  interim  system  until  the  completion  and  installation  of  the 
QA/QC  portion  of  CAMS.  Program  settings  naay  vary  from  MAJCOM  to  MAJCOM.  The 
default  set  depend  on  the  MAJCOM  guidance. 

Function.  QANTTAS  assists  QA  management  in  tracking  information  for  seven  high 
interest  areas  of  QA.  Efficiencies  are  gained  in  the  inspection/reporting  process;  the  ability  to 
identify  negative  trends,  and  greater  control  of  product  improvement  related  effons. 

The  QANTTAS  system  includes  the  functionality  of  PEAP  plus  automated  tracking  in 
other  areas  including  product  improvement.  PEAP  was  used  as  a  Actional  prototype  for 
inspection  tracking,  trend  identification  and  analysis,  therefore  it  is  assumed  that  all  functions 
described  in  PEAP  are  included  in  QANTTAS. 

An  inspection  is  identified  in  two  ways;  the  inspection/evaluation  category  or  type  of 
inspection  and  the  Type  of  Event  Code  (TEC)  which  details  what  kind  of  work  was  inspected. 
There  are  S  inspection/evaluation  categories  in  QANTTAS: 

1.  Technical 

2.  Personnel:  Personnel  Evaluation,  Team  Evaluation,  Evaluator  Proficiency  Evaluation 

3.  Special 

4.  Management 

5.  Safety;  Detected  Safety  violations,  technical  data  violation,  unsafe  condition  report. 
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The  following  highlights  the  Personnel  Category: 

Personnel.  Evaluation  of  maintenance  personnel  while  performing  a  maintenance 
action.  Within  this  category  are  three  evaluation  types: 

1.  Personnel  Evaluation  (PE).  Evaluation  of  the  person  performing  a  task  or  supervising 
the  performance  of  a  task. 

2.  Team  Evaluation  (TM).  Persormel  performing  maintenance  as  a  team  member  are 
evaluated  together  and  the  overall  result  is  reported  as  TM.  Each  individuals 
performance  participation  is  recorded  as  type  PE. 

3.  Evaluator  Proficienc  Evaluation  (EPE).  The  EPE  serves  as  a  cerdficadon  process  for 
people  performing  evaluadons.  To  qualify  as  an  evaluator,  one  must  be  observed 
while  p^otming  a  technical  inspecdon  and  personnel  evaluadon. 

Type  of  Event  (TEC)  codes  are  assigned  by  base  level  QA  managers.  The  first  two 
digits  usually  correspond  with  the  Work  Unit  Cjode  (i.e.,  23  =  Engines).  The  third  digit 
denotes  the  specific  event  (Le.,  23A  =  Engine  final  major  maintenance). 

QANTTAS  can  provide  a  listing  of  inspecdons  broken  down  on  many  criteria.  For 
example,  an  individual’s  inspecdon  history  can  be  called  up  by  entering  the  person’s 
idendficadon  number. 


SALLY 

SALLY  is  described  in  the  Automated  maiiuenance  standardization  and  evaluation 
program  report,  version  3.3  and  3.6,  September  15,  1988.  The  report  states  that  it  should 
work  well  for  any  quality  control  fiincdon  in  all  categories  of  Air  Force  Communicadons 
Command  (AFCC)  units.  SALLY  has  been  in  use  by  the  1913CG  quality  control  fimcdon 
since  January  1986.  The  system  satisfies  all  requirements  of  AFCCR  66-9,  so  it  should  be 
able  to  stand  alone.  SALLY  is  menu  driven. 

The  functions  for  SALLY  are: 

1.  Provides  data  bases  for  all  inspecdons  and  evaluations  required  by  AFCCR  66-9, 
chapter  4. 

2.  It  will  produce  yearly,  monthly,  weekly,  or  anything  in  between  schedule  of 
inspecdons  and  evaluations  due  with  the  date  they  are  to  be  accomplished. 
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3.  It  will  track  report  suspense  and  provide  a  list  of  those  reports  having  passed  a 
suspense  date.  This  is  done  for  reports  due  in  to  or  due  out  of  the  Quality  Control 
section. 

4.  Using  menu  driven  format,  it  will  prompt  an  inspector  through  each  phase  of  creating, 
printing,  and  following  up  on  all  requir^  inspection  or  evaluation  reports. 

5.  Using  menu  driven  format,  the  trend  analysis  reviews  all  discrepancies  previously 
entered  via  the  inspection  generation  option  of  the  program.  Trend  status  repons  are 
generated  that  when  analyzed  will  provide  data  on  deficiency  trends  by  tracking  codes, 
work  center,  type  of  inspection,  type  of  equipment  and  date.  (The  repon  does  not  say 
what  the  tracking  codes  are,  could  be  tied  to  individuals). 

Four  data  bases  are  provided:  technical  inspections,  personnel  evaluations,  activity 

inspections,  and  special  inspections. 

Options  Listed  on  Main  Menu  are: 

1.  Inspection  Data  Base  Maintenance*  maintain  or  update  data  base.  Input  data  at  this 
option. 

2.  Periodic  Inspection  Schedules-  when  user  enters  a  date,  system  produces  inspection 
schedules  to  screen  or  printer  for  technical,  special  and  activity  inspections  and 
personnel  evaluations. 

3.  Daily  Repon  Suspense-  produces  list  of  inspections  that  were  not  entered  on  date  due. 

4.  Generate  Maintenance  Standardization  and  Evaluation  Program  (MSEP)  Inspection 
Reports-  produces  initial  and  follow-up  inspection/evaluation  reports  in  the  AF  Form 
2419/2420. 

5.  Analyze  MSEP  Trends-  reports  sorted  and  selected  so  that  it  is  easier  for  quality 
control  inspector  to  analyze  deficiencies  trends.  (Does  not  indicate  whether  personnel 
evaluations  as  well  as  equipment  inspections  can  be  analyzed). 

Information  Included  in  Reports  are: 

1.  Technical,  Special  and  Activity  Inspections-  information  on  equipment  only,  no 
personnel  information. 

2.  Personnel  Evaluation-  Name,  employee  number,  last  evaluation,  next  evaluation,  task 
evaluated,  evaluation  type  (initial^oUow  on/special),  inspector  (name  and  rank)  and  a 
summary  of  the  evaluation.  Summary  of  evaluation  includes  errors  noted,  task  rating 
(satisfactory  or  unsatisfactory),  and  any  additional  comments. 
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PMEL  Automated  Maintenance  System  (PAMS) 

The  PMEL  operates  differently  than  other  AF  maintenance  operations.  They  use  the 
PMEL  Automated  Maintenance  Systems  (PAMS),  a  form  of  CAMS  specifically  created  for 
PMEL.  A  part  of  PAMS  is  the  Quality  Verification  Inspection  (QVI)  Level.  The  QVI  Level 
is  an  Air  Force-wide  method  for  determining  who  gets  their  work  inspected  and  when  it  gets 
inspected  within  PMEL.  Also,  PAMS  seems  to  monitor  work-time  and  work  quality  closer 
than  the  CAMS  system. 


Aerospace  Ufe  Support  Management  System  (ALSMS) 

ALSMS  is  used  exclusively  in  life  support  specialties  in  MAC.  The  purpose  of 
ALSMS  is  to  automate  record  keeping  for  the  life  support  equipment  inspections.  The  system 
alerts  the  supervisory  personnel  within  life  support  when  a  piece  of  equipment  is  up  for 
inspection.  Data  is  stored  for  the  past  six  inspections  on  each  piece  of  equipment. 

Inspections  are  conducted  on  the  equipment  itself.  No  individuals  are  inspected  or  associated 
with  the  equipment  being  inspected.  The  database  is  local,  within  the  shop.  It  is  a 
FOXBASE  software  system. 
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SUMMARY  AND  CONCLUSIONS 


Summary 

The  results  of  this  study  are  summarized  in  the  matrix  presented  as  Figure  2  in  this 
section.  The  following  summarizes  the  findings  displayed  in  die  matrix. 

Accessibility.  The  accessibility  of  data  is  a  key  factor  determining  if  data  can  be  used 
for  validation  of  selection  criteria.  Paper-and-pencil  records  kept  in  personnel  files  would  be 
very  difficult  to  access.  All  of  the  systems  listed  in  the  matrix  are  automated  and  have  data 
bases.  Therefore,  the  data  are  adequately  obtainable.  The  following  listing  is  more  of  an 
informational  tool  of  how  we  would  access  the  data,  rather  than  whether  it  is  accessible  or 
not. 


Owner.  The  owner  of  the  systems  on  which  the  data  are  contained  is 
MAJCOM  for  CAMS  and  ACTS.  The  remainder  of  the  procedures  are  owned  by  specific 
commands  or  specialties:  TAC  and  MAC  own  PEAP;  USAFE,  PACAF,  and  AAC  own 
QANTTAS;  AFCC  owns  SALLY,  and  Air  Force  PNSL  owns  PAMS. 

Location.  For  all  of  the  systems  the  location  of  the  data  is  the  installation  level 

or  lower. 


Storage.  The  method  of  data  sttnage  varies  in  the  different  systems.  The  data 
for  CAMS  is  stored  on  a  mainframe.  PAMS  is  stored  on  a  local  mainfirame.  ACTS  is  stored 
in  a  PC  network.  PEAP,  QANTTAS  and  SALLY  are  all  PC-based  software. 

Data  Domain.  The  data  domain  indicates  the  mission,  occupation,  and  echelon 
associated  with  the  data. 

Mission.  The  mission  is  the  purpose  for  the  operational  system.  The  mission 
for  AOTS  is  to  automate  and  enhance  OJT  procedures  and  activities  for  all  AFSs  which  have 
formal  OJT  functions.  CAMS  mission  is  to  provide  an  automated  maintenance  management 
system  for  the  base-level  Air  Force  maintenance  community.  PEAP,  QANTTAS,  and 
SALLY  are  to  automate  the  maintenace  system  and  analyze  trends  in  equipment  and 
personnel  for  all  the  maintenance  AFSs.  The  mission  for  QVI  in  PAMS  is  to  have  a 
standardized,  stringent  inspection  system  for  PMEL. 

Occupation.  The  occupation  is  the  specific  AFS,  group  of  AFSs,  or  career 
field  associated  with  the  operational  system.  CAMS,  PEAP,  QANTTAS,  and  SALLY  are  for 
all  maintenance  AFSs.  PAMS  is  PMEL  specific.  AOTS  is  for  use  with  all  AFSs  which 
conduct  OJT  programs. 
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Echelon.  Echelon  is  the  level  at  which  data  are  collected  and  entered  into  the 
system.  The  data  may  be  entered  at  the  individual,  workcenter,  unit,  or  organization  level. 

All  the  systems  included  in  the  matrix  have  data  entered  at  the  workcenter  level. 

Participants.  In  order  to  validate  another  measurement,  the  performance  indicators  we 
select  must  be  tied  to  an  individual,  not  a  team  or  a  unit.  CAMS,  ACTS,  SALLY,  and  PAMs 
contain  data  on  the  individual,  while  PEAP  and  QANTTAS  may  contain  data  on  the 
individual,  team,  or  a  member  of  a  team.  It  poses  a  problem  when  the  team  or  one  member 
of  a  team  is  measured  on  a  team  task  because  it  is  difficult  to  tie  the  perfcnmance  to  an 
individual. 

Item  Measured.  The  item  measured  is  the  item  or  function  on  which  the  measure  is 
taken.  Within  CAMS,  each  of  the  procedures  included  in  the  matrix  measures  different 
things.  The  OJT  measures  JQS  completion,  OJT-Supervisory  Ratings  are  measures  of  job 
performance,  QA/QC  is  the  number  of  inspectitMi  discrepancies,  and  CDC  is  the  score  on  the 
course  exam. 

AOTS  also  has  different  types  of  measures  in  each  of  the  procedures.  Task 
Qualfficadons  is  a  record  of  certification  on  JQS.  Knowledge  Evaluation  is  measure  of 
knowledge  about  task  performance.  Performance  Evaluation  is  an  off-line  evaluation  of 
performance  on  evaluator  selected  or  system  sp^ified  tasks. 

PEAP,  QANTTAS,  SALLY,  and  PAMS  are  all  measures  of  inspection  discrepancies. 
The  type  of  inspections  vary  in  all  of  the  procedures  (i.e.,  over-the-shoulder,  task,  etc.). 

Associated  Data.  The  associated  data  is  an  informational  parameter  to  help  determine 
what  other  information  can  be  attached  to  the  data.  This  varies  between  all  of  the  measures, 
but  it  is  not  a  consideration  in  choosing  t^propriate  measures. 

Procedure.  The  procedure  is  die  method  used  to  obtain  the  data.  Some  common 
procedures  are  observation,  written  and  oral  tests,  over-the-shoulder  and  post-repair 
inspections.  CAMS  JQS  and  AOTS  JQS  use  several  different  procedures  including 
observation,  written  tests,  interviews,  and  hands-cm  tests.  CAMS  supervisory  ratings  and 
SALLY  are  based  on  observations  only.  The  QA/QC  related  measures  (i.e.,  PEAP, 
QANTTAS,  QVI  Levels,  and  CAMS  QA/QQ  use  observation  and  post-repair  inspections. 
CAMS  QA/QC  also  uses  interviews.  Several  measures  use  written  test:  CAMS  CDC  and 
AOTS  Knowledge  Evaluation.  The  AOTS  Performance  Evaluadon  is  by  off-line  testing  using 
an  oral  test  guide  and  a  performance  evaluation  checklist. 

Recording  Method.  All  of  the  procedures  have  on-screen  forms  on  which  the  data  are 
recorded.  CAMS  QA/QC,  PEAP,  and  QANTTAS  also  use  supplementary  Air  Force  forms. 
AOTS  Knowledge  Evaluation  and  Performance  Evaluation  have  the  option  of  using  an  optical 
scan  form  to  enter  the  data  into  die  computer  and  automated  scoring. 
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Type.  All  of  the  procedures  aie  subjective  measures  except  the  measures  involving 
tests;  CAMS  CDC  and  AOTS  Knowledge  Evaluation.  The  QVI  Levels  have  objective  and 
subjective  components.  The  inspections  themselves  are  subjective,  but  the  QVI  Level 
changes  are  objective. 

Research  Question.  Each  of  the  procedures  have  individual  research  questions.  This 
information  indicates  the  reason  the  procedure  was  originally  created  or  its  purpose.  This 
information  is  helpful  in  understanding  the  content  of  the  procedure. 

Fidelity  of  Data.  The  fidelity  of  the  data,  including  reliability  and  content- validity,  is 
listed  on  the  matrix  as  a  parameter  because  of  the  importance  of  this  information  in 
evaluating  a  measure.  We  were  unable  to  determine  fidelity  for  any  of  the  measures. 

Timing.  AOTS  Performance  Evaluation  and  QVI  Levels  are  the  only  measures  which 
are  unannounced  all  of  the  time.  The  CAMS  CDC  and  the  AOTS  JQS  are  both  announced, 
and  the  remainder  of  the  measures  are  announced  and  unannounced  in  different  situations. 

Bias.  Several  of  the  procedures  measure  tire  entire  population  (i.e.,  for  CAMS  the 
JQS,  supervisory  ratings,  and  CDC,  and  for  AOTS  die  JQS  and  Knowledge  Evaluations). 

The  CAMS  QA/QC,  AOTS  Performance  Evaluation,  and  SALLY  are  always  random.  PEAP 
and  QANTTAS  are  usually  random,  but  not  always.  Sometimes  the  inspections  are  based  on 
past  trends  and  in  other  cases  the  supervisor  selects  the  best  perscHi  in  his  shop  to  perform  the 
task  being  evaluated.  The  rules  for  selection  of  who  is  to  inspected  by  QA  seem  to  differ 
between  bases.  The  QVI  Level  is  unique  in  that  the  inspections  are  randomly  chosen  within 
the  constraints  of  the  current  QVI  level  of  the  individual. 

Current  Use  of  Dam.  The  current  use  of  the  data  varies  from  procedure  to  procedure. 
This  information  is  helpful  in  understanding  the  operational  use  of  the  procedures  and  their 
current  status. 


Table  1  indicates  the  extent  the  operational  systems  described  in  this  paper  are  used  in 
the  five  AFS  of  interest  The  predicted  use  of  AOTS  by  the  specialties  is  also  displayed. 
CAMS,  PEAP,  and  QANTTAS  are  used  in  each  of  the  specialties  except  PMEL.  PAMS  is  a 
form  of  CAMS  specifically  created  for  PMEL,  therefore  this  specialty  does  not  use  CAMS. 
SALLY  is  used  only  in  AFCC  specialties.  ALSMS  was  created  for  Aircrew  Life  Suppon, 
and  that  specialty  is  its  only  user.  Unlike  the  other  systems,  PEAP  is  used  by  the  QATQC 
personnel  to  determine  the  trends  within  the  indicated  specialties,  not  by  the  individuals 
within  the  specialties. 
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Table  1.  Operational  Systems  by  Air  Force  Specialty  (AFS) 


AFS 

CAMS 

AOTS 

PEAP 

QANTTAS  SALLY*  PAMS  ALSMS 

Aerospace  Propulsioo 

Specialist  (4S4X0) 

X 

✓ 

X 

X 

Communication  and 

Navigation  System 

Specialist  (4SSX2) 

X 

✓ 

X 

X 

Aerospace  Ground 

Equipment  Mechanic 
(454X1) 

X 

✓ 

X 

X 

Precision  Measuring 

Equipment  Latxmtory 

Specialist  (324X0) 

✓ 

X 

Aircrew  Life 

Support  Specialist 
(122X0) 

X 

✓ 

X 

X 

X  Cunent  use 

/  Proposed  use 

•  Used  only  in  AFOC 

Conclusions 

It  is  clear  from  the  summary  of  the  matrix,  as  shown  in  Hgure  2,  that  we  did  not 
discover  any  Air  Force-wide  job  performance  measures  at  the  operational  level  that  could 
function  as  surrogate  measure  for  JPMS  developed  HOPT.  The  automated  systems  are  for  the 
most  part  MAJCOM  specific.  There  are  no  central  Air  Force-wide  data  bases  associated  with 
the  systems  reviewed.  Data  bases  are  at  MAJCOM,  installation,  ot  unit  level.  However  there 
are  a  few  systems  that  have  potential  for  use.  These  are  the  PAMS,  CAMS,  and  AOTS. 

PAMS.  The  PMEL  AFS  uses  QVI  Level  within  PAMS  to  determine  who  gets  their 
work  inspected  and  when  the  inspection  takes  place.  Although  used  by  PMEL  as  a  quality 
control  fiincdon  for  equipment  repair  and  calibration,  this  measure  is  an  indicator  of  job 
performance.  It  measures  the  ability  to  perform  worit  without  errors.  The  inspection  is  on 
individuals  and  is  always  unannounced.  The  procedure  is  systematic  in  determining  which 
items  to  inspect.  The  inspection  itself,  although  very  structured,  is  subjective,  but  the  system 
used  for  changing  an  airman’s  QVI  Level  is  very  structured  and  objective.  The  information  is 
automated  and  stored  at  the  unit  level. 


32 


CAMS.  Although  CAMS  was  developed  to  automate  maintenance  procedures  it  also 
implements  the  OJT  requirements  in  Air  Force  Regulation  50-23.  CAMS  can  be  used  to 
automate  OJT  procedures.  The  data  base  contains  a  record  of  the  progress  of  an  airman 
during  OJT.  In  addition,  CAMS  has  supervisor  ratings  of  OJT  performance  (satisfactory  or 
unsatisfactory),  CDC  course  exam  scores,  and  AKT  scores.  When  the  QA/QC  module  is 
implemented  the  data  base  will  include  discrepancy  data  and  the  results  of  personnel 
evaluations. 

CAMS  contains  data  that  meets  the  criteria  for  a  candidate  surrogate  measures  of  job 
performance.  Although  not  Air  Force-wide,  die  data  base  is  at  the  installation  level  with 
access  at  the  MAJCOM  level.  It  contains  a  record  of  OJT  performance  which  is  based  on 
observation,  interview,  and  hands-on  testing;  has  supervisor  ratings;  and  contains  CDC  and 
AKT  scores. 

AOTS.  AOTS  was  designed  to  be  in  strict  compliance  with  Air  Force  Regulation  50- 
23.  The  system  was  designed  specifically  to  improve  the  procedures  for  OJT.  A  fully 
operational  AOTS  could  provide  the  research  community  with  a  valuable  source  of 
operational  measures.  What  makes  AOTS  significant  is  that  it  focuses  on  job  position 
qualification,  whereas  OJT  focuses  on  job  task  qualification.  The  system  data  base  will 
contain  the  results  of  performance  evaluations  accomplished  for  pre-training,  post-training, 
and  for  quality  control  purposes.  The  job  performance  evaluations  are  conducted  off-line 
using  system  generated  evaluation  guides.  There  is  a  GO/NO  GO  rating  for  each  step  in  the 
procedure  being  evaluated.  When  the  results  are  entered  into  the  system,  the  system  evaluates 
the  overall  procedure.  Implications  are: 

•  Pre-training  evaluations  can  be  an  indicator  of  how  well  the  technical  school 
prepared  the  airman  for  the  job. 

•  Pre-training  and  post-training  evaluations  can  indicate  of  how  well  the  AKT 
screened  personnel  for  assignment  to  the  AFS. 

•  Post-training  and  quality  control  evaluations  can  be  assessed  as  surrogate  JPMS 
measures. 

The  AOTS  data  base  will  also  contain  the  record  of  OJT  progress  (JQS/STS)  and  the  results 
of  evaluations  of  knowledge  of  how  tasks  are  performed.  Since  any  measure  of  job 
performance  should  take  into  account  the  total  person  in  the  total  job,  these  data  could  be 
used  as  a  composite  part  of  a  surrogate  measure  of  job  performance.  Also,  SKT,  CDC,  and 
EPR  scores  could  be  a  pan  of  the  composite  measure. 

All  of  the  data  in  the  AOTS  data  base  are  keyed  to  the  individual  airman.  The 
expansion  plan  calls  for  a  fully  networked  PC-based  system.  A  networked  system  would 
permit  ready  access  to  the  data. 
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GLOSSARY 


Abbreviations 

AAC 

Alaskan  Air  Command 

AFCC 

Air  Force  Communications  Command 

AFHRL 

Air  Force  Human  Resources  Laboratory 

AFJQS 

Air  Force  Job  Qualification  Standard 

AFLMC 

Air  Force  Logistics  Management  Center 

AFMPC 

Air  Force  Military  Personnel  Center 

AFRES 

Air  Force  Reserves 

AFS 

Air  Fbrce  Specialty 

AFSC 

Air  Force  Systems  Command 

AGE 

Aerospace  Ground  Equipment 

AKT 

Apprentice  Knowledge  Test 

ALS 

Aircrew  Life  Suppon 

ALSMS 

Aerospace  Life  Support  Management  System 

ACTS 

Advanced  On-the-Job  Training  System 

APR 

Airman  Performance  Report 

ATC 

Air  Training  Command 

BTS 

Base  Training  System 

CAMS 

Cote  Automated  Maintenance  System 

CDC 

Career  Development  Course 

CJQS 

Command  Job  Qualification  Standard 
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CMA 

Con^leted  Maintenance  Action 

CMI 

Convicted  Maintenance  Inspection 

CSI 

Completed  Supervisory  Inspecdtm 

DCM 

Deputy  Commander  for  Maintenance 

EPE 

Evaluator  Proficiency  Evaluation 

EPR 

Enlisted  Performance  Report 

FID 

Field  Training  Detachment 

HOPT 

Hands-On  Performance  Test 

JKT 

Job  Knowledge  Test 

JPM 

Job  Performance  Measurement 

JPMS 

Job  Performance  Measurement  System 

JQS 

Job  Qualification  Standard 

MAC 

Military  Airlift  Command 

MAJCOM 

Major  Command 

MSEP 

Maintenance  Stardardization  and  Evaluation  Program 

NCO 

Noncommissioned  Officer 

OJT 

On-the-Job  Training 

CMC 

Occupational  Measurement  Center 

PACAF 

Pacific  Air  Force 

PAMS 

PMEL  Automated  Maintenance  System 

PE 

Personnel  Evaluation 

PEAP 

Personnel  Evaluation  and  Analysis  Program 
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pm 

Product  Lnprovement  Branch 

PMEL 

Precision  Measuring  Equipment  Laboratory 

QANTTAS 

Quality  Assurance  Tracking  and  Trend  Analysis  System 

QA/QC 

Quality  Assurance/Quality  Cmitrol 

QVI 

Quality  Verification  Inspection 

SAC 

Strategic  Air  Command 

SALLY 

Automated  Maintenance  Standardization  and  Evaluation  Report 

SE 

Supervisory  Evaluation 

SKT 

Specialty  Knowledge  Test 

STS 

Specialty  Task  Statements 

SME 

Subject  Matter  Expert 

TAC 

Tactical  Air  Command 

TAP 

Tactical  Air  Forces 

TE 

Task  Evaluation 

TEC 

Type  of  Event  Code 

TTDB 

Training  Technology  Data  Base 

TPDC 

Training  Performance  Data  Center 

UGT 

Upgrade  Training 

USAFE 

United  States  Air  Force  in  Europe 

WAPS 

Weighted  Airman  Promotion  System 

WTPT 

Walk-Through  Performance  Test 
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APPENDIX  A 


INTERVIEW  PROTOCOL 

We  are  ficom  the  Human  Resources  Research  Organization  (HumRRO).  We  are  under  contract 
to  the  Air  Force  Human  Resources  Laboratory  to  examine  job  performance  measures  currently 
used  in  the  Air  Force.  The  purpose  of  this  intmview  is  to  get  as  much  information  as  possible 
on  current  formal  and  infonnal  methods  used  by  supervison  and  others  to  assess  an  individual’s 
job  performance. 

1.  Do  you  Imow  of  any  individual  job  performance  measures  that  are  routinely  collected? 

I 

2.  Are  other  records  or  information  collected  that  indicate  job  performance?  Are  these  used 
as  performance  measures  in  any  way? 

3.  What  are  the  names  of  these  measures? 

For  each  measure  named  ask  the  following  questions: 

4.  (If  not  indicated  by  name  of  measure)  What  type  of  measure  is  this  (i.e.,  Rating  form. 
Skill  Knowledge  Test  (SKT),  etc.)? 

5.  Where  is  this  daita  kept? 

6.  In  what  form  is  it  truuntained  (e.g.,  computer,  hard  copy)? 

7.  Who  controls  the  access  to  this  data? 

8.  Would  the  data  be  accessible  to  the  research  community  (e.g.,  HRL)? 

9.  How  widely  used  is  this  measure?  Is  the  measure  Air  Force-wide  or  is  the  use  limited 
to  specific  installations  or  unit  levels,  ere.? 

10.  (If  the  junction  or  task  that  is  being  measured  has  not  been  indicated  yet)  What  is  the 
function  or  task  that  is  used  in  the  measure?  For  example,  is  there  a  certain  procedure 
that  is  rated,  or  is  it  a  measure  of  units  of  production,  or  number  of  mistakes? 

11.  Who  are  the  participants  in  this  measure?  What  is  the  rank,  time  in  service  (e.g.,  first 
enlistment)  of  the  Airmen  on  which  this  measure  is  used? 

12.  (If  the  procedure  has  not  been  indicated  yet)  What  is  the  procedure  used  to  get  this 
information  on  the  Airmen?  Are  they  observed,  is  it  a  paper  and  pencil  test,  or  is  there 
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some  other  way  to  coilea  the  measurement  information  (Le.,  electronic  transmission)? 

13.  How  is  the  information  recorded?  Is  it  a  checklist,  a  special  form,  a  standard  Air  Force 
form,  or  is  the  informadon  on  other  records  previously  recorded? 

14.  What  is  the  intended  purpose  of  collecting  this  data  (i.e.,  training  effecdveness,  unit 
readiness,  promotion,  etc.)? 

15.  Do  you  use  it  for  any  other  purpose? 

16.  Does  the  airman  take  this  test  more  than  once?  What  are  the  locations?  What  is  the  time 
difference  between  testing? 

17.  (If  a  subjective  measure)  Who  gives  the  ratings?  Does  more  than  one  person  administer 
this  measure? 

18.  What  job  or  task(s)  is  this  measure  aimed  at  tapping?  Are  there  any  other  performance 
measurement  systems  measuring  similar  tasks? 

19.  Are  the  airmen  told  in  advance  that  they  will  be  assessed?  If  so,  how  far  in  advance? 

20.  How  often  is  the  measure  administered?  Is  it  given  once,  throughout  a  certain  job,  or 
periodically  throughout  career? 

21.  If  given  periodically  (e.g.,  throughout  a  training  session,  or  throughout  the  time  in  the  Air 
Force)  at  what  point(s)  is  it  given? 

22.  Who  is  assessed  by  this  measure  (e.g,  a  few,  a  group,  or  all  of  the  Airmen)?  How  is  it 
determined  how  many  will  participate?  How  is  it  determined  who  will  participate? 

23.  What  is  done  with  the  data  once  it  is  collected?  Who  receives  the  data?  How  is  it  used? 
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